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ABSTRACT 

The study investigated whether.=^bool administrators 
(n=128) implicitly perceive the "types" of evaluators proposed by 
Meltsner (1976) when they consider evaluative information. One 
s^mulated^evaluation report wa^ written to represent each evaluator, 
differentiated from one another by their norms, expectations, 
motivations, and training. Two pilot studies 'were conducted, to assess 
the validity of the simulation materials. Graduate students were 
asked to read all four reports to identify which "type" each report 
represented, and isolate which report features led to their 
decisions. Administrators w6re then asked. to indicate: (1) the 
similarity of the report to their school system reports; (2) 
evaluator fairness; (3) technical competence of the evaluator; ahd 
(4) evaluator awareness that political consideratioxHS^ometimes 
affect program decisions. Finally, administrators were asked to make 
a policy recommendation regarding the program's future. Results 
indicate that administrators do perceive evaluator types When they 
consider evaluative information presegnted in reports. However, the 
results suggest that administrators are more attentive to the 
technical merit of reports than they are to the evaluator 's political 
sensitivity, as indicated by the homogeneity of the subjects' 
perceptions of the political sensitivity of the various types of 
reports^ This attention to the technical has been suggested in 
earlier research. (Author/PN) 
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ABSTRACT 



The study investigated , ^i^/hether administrators implicitly 
perceive the "types" of evaluators proposed by Meltsner (1976) 
when they consider evaluative information. One simulated 

CI 

evaluation report was written to represent each evaluator type. 
Two pilot studies were conducted to assess the validity of the 
simulation materials. The results suggest that administrators 
may implicitly, employ the typology. The ecological validity of 
the results was also empirically investigated . 
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•Adminlstratoi^s^ Perceptions of Evaluator Types Page 2 

The literature on program evaluation contains numerous 
desultory comments reiar^in'fe the extent to which administrators 
use evaluative information (Thompson & King, 1981, pp. 3-4). 
For example, Cronbach et al. (1980, p. i) lament that 
"commissioners of evaluations comjJlaih that the messages from 
evaluation are not very useful, while, evaluators complain that 
the messages are not use^l." Stake (1973, p. 314) has gone so • 
far as to say that "we d6 not know whether or not evaluation is 
going to contribute more to the problems of education or to the 
solutions." Alkin and Daillak (1979, p. 41) -sum up the 
situation by noting that "there have been great hopes for 
evaluation, not only among evaluators'' themselves, but also 
among other educators, elected officials, and the public. Yet 
these hopes have dimmed." ^ ^ 

Unfortunately, the non-use of evaluative information, when 
that use would be appropriate, respresents an enormous waste of 
effort (Datta, 1979, /p. 22), an enormous waste of money 
(Pattpn, 1978, p. 12), and results in less than optimal service 
being provided to program qlients. This situation is tragic 
because, as Wise (1980, p. 16) notes, "no one else is given the 
resources and time to question, observe, assess, weigh, probe, 
and reflect that the evaluator is given." Con sequentl y , sever a 1 
researchers have attempted to identify the factors which do or 
do not affect use levels. 
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Several factors "whicK apparently do not effectively 

promote Utilization have' been identified. For example, it is 

now clear that the numerous evaluation "models" 

(cf. Stufflebeam, Foley, Gephart, Cuba, Hammond, Merriman & 

Provus, 1971) are not panaceas for promoting use. The models 

have been useful ^ to an emerging disc ipline trying to 

conceptualize its role, but the models are not very useful 

guides (nor perhaps were they meant t^ be) for evaluation 

practice. Thus Airasian (197U, p. 148) notes that "the many 

evaluation models advanced in recent years suggest what 

decisions need to be made but not how they should be made." 

Based on his case studies of evaluative practice , Alkin (1979, 

p. 7, emphasis in original) found that "none of the five cases 

involved ,the application of a formal evaluation model ." As 

Brown (1980, p. 4) summarized the matter: 

For a time,, it was hardly respectable to 
be an evaluator without hav ing your own 
model . You at least had to be a 
disciple of a pr oponent _of " a new mo^del 
that was on the "cutting edge" in order 
to maintain some semblance of 
self-esteem. It is interesting to 
observe that there were very few wounds 
inflicted by that "cutting edge." 

It has also become clear that the methodological aspects 

of evaluations are not the primary determinants of use. In 

fact, there is some empirical evidence (Brown & Newman, in 

press) that inclusion of inferential statistics in evaluation 

reports even tends to lessen use. Although this result may 
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have partially been a function of the manner in which* the 

inferential statistics were presented (Thompson, 1981a, p. 6), 

nevertheless, a growing body of evidence suggests that "the 

major barriers" to .successful evaluation are not technical and 

* 

methodological, though these are certainly important and worthy 
of further effort, but are rather the structural constraints 
and requirements and the interpersonal relationships^ which 
characterize the evaluation endeavor" (Gur el , ' 1 975 , pp. 27-2-8). 

One of the most critical determinants of evaluation use 
discussed in the literature is what Pa^tton ( 1 978) has termed 
"the personal factor." As Cronbach et al. (1980, p. 6) 
summarize, "nothing makes a larger difference in the use of 
evaluations than the personal factor--the interest of officials 
in-learning from the evaluation and the desire of the evaluator 
tQ get attention for what he knows." Evaluators who are 
sensitive to this factor will respond in two ways. 'First, 
following the suggestion of Patton (1978), evaluators will 
identify* the evaluation's relevant information users, and they 
will continually work on affecting utilization throughout the 
course of the evaluation. Second, evaluators will also attempt 
to establish close working relationships with evaluation 
clients. 
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One theoretical framework- regarding these interpersonal 
relationships has been offered by Meltsnef .( 1976), who argues! 
that ev'aluators are differentiated from each other by their 
norms, expectations, motivations, and training. During the 
early 1970's Meltsner and a colleague interviewed 116 federal 
policy analysts. The data from these interviews suggested that 
evaluators might be differentiated from each other primarily by 
how much analytical skill they possess and by how much 
political skill they have developed. This conceptualization 
led to a' fourfold typology of evaluators.^ 

^•Entrepreneurs" are characterized" as having both good 
political and good technical skills. "Technicians" are 
characterized as having good technical skills and less 
satisfactory political skills. "Politicians" are characterized 
as having good political but weak technical skills. 
"Pretenders" are characterized as being weak on both skill 
dimensions . 

\ 

Some research suggests that evaluators implicitly perceive 
themselves in terms of Meltsner' s conceptual model (Thompson, 
1980a, 1980b). However, it has not yet been demonstrated that 
the model helps to -explain how school administrators perceive 
evaluators. The .research reported here vTas' conducted to 
investigate this possibility. 
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Specif icajlly , the study - addressed three questions. Do 
school administrators implicitly use an evaluator typology, 
(Meltsner, 1976) when they^ consider evaluation reports? Do 
these perceptions in*fluence the policy recommendations which 
the administrators offer? Finally, how do perceptions of 
report techn^ical quality-^nd political sensitivity interact 
with each. other and other variables? 

Method • 

Subjects ^ I 

The subjects (n = 128) in the study were principals in 

public school systems in the Southern United States. Only 

principals were included to avoid any confounding of results 

that might have been introduced by variations in administrative 

perspective. Furthermore, principals form an important 

evaluation audience which has not been consider*ed frequently 

enough in previous evaluation research. There have been use 

♦ 

studies involving principals as subjects (GranviMe,. 1977; 
Hamilton, 1980), but the number of such studies does not seem 
proportional to the influence which principals may exert over 
school decision-making {Lipham^;^ 1980, p. 3). 
Materials 

The study used simulated evaluation reports for three 
reasons : first , . evaluators ' contact with pr incipals' 
frequently takes the form of such written reports; second, the 
variables of interest could be carefully controlled; and 
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third, earlier simulation studies suggest the value of thia 
approach (see, for example, Newman, Brown 4 Braskamp, 1980). 
While it is true that principals' perceptions of an evaluator 
typology will be evidenced primarily in their react ion/ te-; 
practicing evaluators, their responses to reports refle(/,irfg 
differences may also provide evidence for the validity oT the 
typology. With this in mind, an evaluation report was wfitten 
as an exemplar of each of the four evaluator pers/ectives 
'proposed by Meltsner (1976). The reports were s/railar in 
coa-tent--they all dealt with the same hypothetical j^ogram--but 
varied in style, e.g., in the amount of technical information 

includ*<|, "and in the number of references to specific 

/ 

individuals. / 

Two pilot studies were conducted in order ' to refine and 
validate the study's simulation materfials. In the first pilot 
study 23 graduate students were taught the "types" and were 
then adked to read all four reports, attempt to identify which 
"type" each report represented, and isolate which Veport 
features led to their decisions. Based upon these results the 
four reports were revised and a second pilot study was 
conducted. Ten different graduate' students from-a different 
university were taught the "types" and were then asked to 
identify which "type" each report represented. The number of 
correct, identifications for the "entrepreneur," "technician," 
"politician," and the "pretender" were, respectively, seven. 



nine, nine, and eight. All four values represent statistically 
significant deviations Jrom expected chance performance 
(£ < .01). The numbers of correct identifications were 
evaluated using a binomial test distributiein with the a priori 
probability of a correct identification being considered .25 
(Freund, 1971). 

« 

Procedure 

Each subject was randomly^ assigned one "type" of report. 

Subjects were mailed the reports, ^sked to answer five brief 

questions, 'and then return the responses in stamped, 

pre-addressed envelopes. Specifically, the subjects were asked 

to indicate how similar the report was "to evaluation reports 

p 

written about programs in my school System." (hereafter labelled 
"similarity"), whether "the report gives me the impression that 
the evaluator was attempting to be fair," whether "the report 
gives me the impression that the evaluator has the technical 
competence to address evaluation issues," and whether "the 
report gives me the impression that the evaluator recognizes 
that political considerations sometimes affect decisions about 
programs." In order to maximize reliability, responses were 
gathered using a nine-interval Likert scale (Thompson, 1981b). 
Finally, the, subjects were asked to make a policy 
recommendation' regarding the program's future. .Participation 
was anonymous and 128 usable responses represented a response 
rate of roughly 40X. 

10 
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Results 

The first research question posed in jthe study asked 
whether school administrators implicitly employ Meltsner's 
(1976) evaluator typology when they consider evaluative 
information. The question was addressed by conducting a 
discriminant function analysis. The four report types 
constituted a nominally-scaled criterion variable. Subjects' 
ratings of report "similarity," fairness, technical quality and 
political sensitivity constituted the predictor variables. One 
statistically significant ( = 26.9, df = 12, £< .0-1) 
discriminant function was identified. 

However, the preponderance of the function's predictive 
efficiency was contained in the technical-quality variable. 
This was indicated in several ways. First, only this variable 
had a statistically significant univariate F-ratio (F = 4.4, 
df = 3,119, £< .01). Second, the variable's structure- 
coefficient (Thompson 4- Frankiewicz, 1979) was -.7; the 
structure coefficients for all the other predictor variables 

were less than an absolute value of .1. 

■ s ♦ 
A numerically higher rating on the technical-quality 
variable represents perception of greater technical merit.. The - 
mean ratings^ for the "technician" and "entrepreneur" " reports 
were, respectively, 7.0 (SD = "2.2) and 6.7 (SD c 2.2). The 
mean ratings for the "politician" and "pretender" reports were, 

r 

11 
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respectively, 5.5 (SD = 2.8) and 5.0 (SD = 2.6). 

The second question posed in the study asked whether 
perceptions of evaluator type influenced admirj^istratpra ' policy 
recommendations. There were no noteworthy variatior/s in polidy- 
recommendations associated with which reports the subjects 
received. However, this result may have been an artifact of 
^ the fact that most of the respondents (86X) reconlmended that 
the portrayed' program be modified to determine whether it could ' 
be made more effective. 

The third question posed in the study asked how* 
perceptions of report technical quality and political^ ^ 
sensitivity interact with each other and with other variables. 
This question was addressed by computing bj.variate correlation 
coefficients. They are presented in Table 1. All the values 
are statistically significant (£ < .01). 

INSERT TABLg J_ ABOUT HERE. 

Discussion ^ 
One distinguishing feature of the research was an effort 
to determine the ecological validity of the results. Subjects 
were asked to indicate whether the study's reports were similar 
to reports produced in their school system. Approximately 61% 
of the subjects agreed that the reports were typical. 
Furthermore, all analyses were reconducted eliminating subjects 
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,who.- felt the reports were atypical. The^ findings were 
unchanged. ?his suggests that the study's results have some- 
generalizability . . . ■ ^ " * 

The results of the analjj^^s addressing' the study's first 
research ^question .suggest that' administrators do perceive 
evaluator types when, they consider evaluative information 
presented in r^orts. However, the results llsuggest that 
administrators are more attentive to the technical pierit of 
repot^tsythan they are to the evaluator's political sensitivity , 
as indicated by the homogeneity of the subjects' perceptioris of 
the political sensitivity ,of the vai^ious types of reports.. 
This "Attention to the technical has been sugested in"" — earlier 
research. In summarizing .a series of simulation studies , ' 
Newman, Braskamp, and Brown ^( 1980) note that evaluation reports 
containing jargon and data--both of which make ' a report more 
technical — were generally rated more.|^usef ul by readers, 
regardless of their field. The failure to find differences on 
the political-sensitivity variable may have been a function of 
the study's design. Administrators may formulate impressions 
of evaluator technical ability on the basis of evaluation 
reports; they may prefer to ' judge' evaluator political 
sensitivity on the basis of personal interactions. This 
^ssibility remains to be expl in future research. 
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The results ^addressing the study's second research 
question are 'arabiguo,us regarding , the sl^udy's theoretical 
underpinnings; but may offer: a valuabre comment on simulation 
research. , The failure of more subjects io offer ^ policy 
recommendation other than ^ program refinement may reflect a 
hesitancy to m^ke "negative" recommendations on the basis of 
brief simulation reports. Simulation researchers confront a 
dilemma. • They can write longer, more detailed simulation 
reports which may better generalize to natural ecologies, but 
which may take more time to administer and may lower response 
ra,tes. Or researchers can write brief reports which inherently 
will not represent very important dynamics affecting 
administrator decisioning (iee Patton, 1978, especially 
p'. 266). This does not mean that simulation research should be 
abandoned, but^ its limits must certainly be recognized. • 

- The results addressing the study's third research question 
are important theoretically. The primary fin,ding is that 
perceptions of technical Equality and political sensitivity are 
reasonably independent (r squa>ed •= .04). This finding is 
no^^if^^thy because it suggests that the two dimensions of 
Meltsner's typology are unrelated, as postulated. 

In summary, the results represent an initial attempt to 
determine whether school administrators perceive evaluators in 
terms of the typology proposed by Meltsher (1976). The results 
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are not fully conclusive, but generally indicate that the model 
may be valid. Thus, the model may provide valuable assistance 
to efforts to understand what Gur'el ( 1975, PP- 27-28) has 
termed the structural and interpersonal constraints on the 
evaluation endeavor. 
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Table 1 
Correlation Coefficients 



Fairness (F) 
Technical 

Quality (T) .68 
Political 

Sensitivity (P) .25 .21 
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